Mems mirror-based extended reality projection with eye-tracking

ABSTRACT

An image projection system includes a first transmitter configured transmit pixel light pulses along a transmission path to be projected onto an eye to render a projection image thereon; a second transmitter configured to generate infrared (IR) light pulses transmitted along the transmission path to be projected onto the eye and reflected back therefrom as reflected IR light pulses on a reception path; a coaxial scanning system arranged along the transmission and reception paths; an eye-tracking sensor configured to generate a retina image of the eye based on reflected IR light pulses, and process the retina image to determine a fovea region location of the eye; and a system controller configured to render the projection image based on the fovea region location, wherein the projection image is rendered with a higher resolution in the fovea region and is rendered with a lower resolution outside of the fovea region.

BACKGROUND

Augmented reality (AR) is a technology that augments physicalenvironments on a mobile device screen by overlaying them with digitalcontent. It adds digital elements to a live view. For example, acaptured piece of an environment is augmented with digital informationthat is superimposed thereon. Thus, digital content is overlaid onto thecaptured piece of the environment to visually provide additionalinformation to a user. The digital content may be displayed on atransparent substrate or display, such as smart eye-glasses, smartcontact lenses, head-up displays (HUDs), and head-mounted displays(HMDs), or projected directly onto a user's retina, as is the case forvirtual retinal displays.

Virtual reality (VR) is a technology that entirely replaces the realworld environment of a user with a computer-generated virtualenvironment. Thus, a user is presented with a completely digitalenvironment. In particular, computer-generated stereo visuals entirelysurround the user. In a VR simulated environment, a VR headset thatprovides 360 degree vision may be used.

A mixed reality (MR) experience combines elements of both AR and VR suchthat real-world and digital objects interact. Here, a real worldenvironment is blended with a virtual one. In addition to theaforementioned technologies, a hololens may be used to provide an MRenvironment to a user.

These technologies, as well as others that enhance a user's senses, maybe referred to as extended reality (XR) technologies.

In order to enhance a user experience in an XR technology, it may bebeneficial to implement eye-tracking to track a direction a user islooking.

SUMMARY

One or more embodiments provide an image projection system that includesa first transmitter configured to generate pixel light pulses andtransmit the pixel light pulses along a transmission path to beprojected onto an eye to render a projection image thereon; a secondtransmitter configured to generate infrared (IR) light pulsestransmitted along the transmission path and to be projected onto the eyeand reflected back therefrom as reflected IR light pulses on a receptionpath; a coaxial scanning system arranged along the transmission path andthe reception path, the coaxial scanning system including at least oneoscillator structure that enables the coaxial scanning system to steerthe pixel light pulses and the IR light pulses in a first scanningdirection and in a second scanning direction according to a scanningpattern; an eye-tracking sensor configured to receive the reflected IRlight pulses from the coaxial scanning system, generate a retina imageof the eye based on the reflected IR light pulses, and process theretina image to determine a location of a fovea region of the eye; and asystem controller configured render the projection image based on thelocation of the fovea region, wherein the projection image is renderedwith a higher resolution in the fovea region and is rendered with alower resolution outside of the fovea region.

One or more embodiments provide a method of projecting an image based onfovea tracking. The method includes transmitting pixel light pulsesalong a transmission path to be projected onto an eye to render aprojection image thereon; transmitting infrared (IR) light pulses alongthe transmission path, the IR light pulses being projected onto the eyeand reflected back therefrom as reflected IR light pulses on a receptionpath; steering the pixel light pulses and the IR light pulses in a firstscanning direction and in a second scanning direction according to ascanning pattern; sensing the reflected IR light pulses received fromthe reception path; generating a retina image of the eye based on thereflected IR light pulses; processing the retina image to determine alocation of a fovea region of the eye; and rendering the projectionimage based on the location of the fovea region, wherein the projectionimage is rendered with a higher resolution in the fovea region and isrendered with a lower resolution outside of the fovea region.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are described herein making reference to the appendeddrawings.

FIG. 1 is a schematic block diagram of an image projection system 100 inaccordance with one or more embodiments;

FIGS. 2A and 2B illustrate two scanning patterns in a 2D scanning planeaccording to one or more embodiment;

FIG. 3A shows MEMS mirror driving waveforms for scanning according toone or more embodiments;

FIG. 3B shows a scanning pattern generated based on the MEMS mirrordriving waveforms shown in FIG. 3A;

FIG. 4A shows MEMS mirror driving waveforms for scanning according toone or more embodiments; and

FIG. 4B shows a scanning pattern generated based on the MEMS mirrordriving waveforms shown in FIG. 4A.

DETAILED DESCRIPTION

In the following, various embodiments will be described in detailreferring to the attached drawings. It should be noted that theseembodiments serve illustrative purposes only and are not to be construedas limiting. For example, while embodiments may be described ascomprising a plurality of features or elements, this is not to beconstrued as indicating that all these features or elements are neededfor implementing embodiments. Instead, in other embodiments, some of thefeatures or elements may be omitted, or may be replaced by alternativefeatures or elements. Additionally, further features or elements inaddition to the ones explicitly shown and described may be provided, forexample conventional components of sensor devices.

Features from different embodiments may be combined to form furtherembodiments, unless specifically noted otherwise. Variations ormodifications described with respect to one of the embodiments may alsobe applicable to other embodiments. In some instances, well-knownstructures and devices are shown in block diagram form rather than indetail in order to avoid obscuring the embodiments.

Further, equivalent or like elements or elements with equivalent or likefunctionality are denoted in the following description with equivalentor like reference numerals. As the same or functionally equivalentelements are given the same reference numbers in the figures, a repeateddescription for elements provided with the same reference numbers may beomitted. Hence, descriptions provided for elements having the same orlike reference numbers are mutually exchangeable.

Connections or couplings between elements shown in the drawings ordescribed herein may be wire-based connections or wireless connectionsunless noted otherwise. Furthermore, such connections or couplings maybe direct connections or couplings without additional interveningelements or indirect connections or couplings with one or moreadditional intervening elements, as long as the general purpose of theconnection or coupling, for example to transmit a certain kind of signalor to transmit a certain kind of information, is essentially maintained.

The term “substantially” may be used herein to account for smallmanufacturing tolerances (e.g., within 5%) that are deemed acceptable inthe industry without departing from the aspects of the embodimentsdescribed herein.

In the present disclosure, expressions including ordinal numbers, suchas “first”, “second”, and/or the like, may modify various elements.However, such elements are not limited by the above expressions. Forexample, the above expressions do not limit the sequence and/orimportance of the elements. The above expressions are used merely forthe purpose of distinguishing an element from the other elements. Forexample, a first box and a second box indicate different boxes, althoughboth are boxes. For further example, a first element could be termed asecond element, and similarly, a second element could also be termed afirst element without departing from the scope of the presentdisclosure.

Embodiments relate to optical sensors and optical sensor systems and toobtaining information about optical sensors and optical sensor systems.A sensor may refer to a component which converts a physical quantity tobe measured to an electric signal, for example a current signal or avoltage signal. The physical quantity may, for example, compriseelectromagnetic radiation, such as visible light (VL), infrared (IR)radiation, or other type of illumination signal, a current, or avoltage, but is not limited thereto. For example, an image sensor may bea silicon chip inside a camera that converts photons of light comingfrom a lens into voltages. The larger the active area of the sensor, themore light that can be collected to create an image.

A sensor device as used herein may refer to a device which comprises asensor and further components, for example biasing circuitry, ananalog-to-digital converter or a filter. A sensor device may beintegrated on a single chip, although in other embodiments a pluralityof chips or also components external to a chip may be used forimplementing a sensor device.

In the field of extended reality (XR) technologies, a sensor may be usedfor eye-tracking to detect and track a direction in which a user islooking. Doing so may allow the XR system to use foveated rendering, atechnique that shifts a high-detailed region of an image to coincidewith a projection onto the fovea. Dynamic foveated rendering follows theuser's focal direction (i.e., a user's gaze) in real-time usingeye-tracking or gaze-tracking and renders a sharp image wherever theuser's retinas are looking rather than at any fixed location.Embodiments disclosed herein are directed to detecting and tracking aneye direction (i.e., a focal direction), and, more particularly, todetection and tracking a fovea position of a user's eye in order tocompensate a scanning operation of a scanning system. Based on thedetected focal direction and/or the detected fovea position, a systemcontroller is configured to adjust one or more system parameters,including: a scanning frequency of a scanning structure, a scanningpattern, a light pulse emission time of an red-green-blue (RGB)projection, and/or a beam width of an RGB projection.

FIG. 1 is a schematic block diagram of an image projection system 100 inaccordance with one or more embodiments. The image projection system 100comprises an eye-tracking system that enables dynamic foveated renderingof the projected image. The image projection system 100 includes an RGBlight unit 10 (i.e., a first transmitter) having plurality of lightsources, including red (R), green (G), and blue (B) monochromatic lightsources (e.g., laser diodes or light emitting diodes). The RGB lightunit 10 is configured to generate red, green, and blue light beams inthe visible light spectrum corresponding to image data to be projectedonto the retina of a user's eye. The RGB light unit 10 is configured totransmit the red, green, and blue light beams towards a scanning system20. Each RGB light pulse may be representative of an image pixel of anRGB image. Each RGB light pulse may comprise any combination of a redlight pulse, a green light pulse, and/or a blue light pulse emittedsimultaneously, including one, two, or three colors in combination atcontrolled intensities according to the desired pixel hue of therespective image pixel. Accordingly, an RGB light pulse may be referredto as a pixel light pulse.

The scanning system 20 has a coaxial architecture in that a transmissionpath is shared with a reception path. In other words, the components ofthe scanning system are used both to transmit light at a user's eye, andto receive reflected light or backscattered light from the user's eye inthe form of sensor data. The scanning system 20 includes a firstmicroelectromechanical system (MEMS) mirror 21, a first beam collimator22 (i.e., a first collimation lens), a second beam collimator 23 (i.e.,a second collimation lens), and a second MEMS mirror 24. The first beamcollimator 22 and the second beam collimator 23 for a relay opticssystem between the two MEMS mirrors 21 and 24 for transmitting lightsbeams (i.e., RGB light beams and infrared (IR) light beams)therebetween. However, the relay optics system is optional such thatfree-space propagation between the two MEMS mirrors 21 and 24 may beused. Additionally, another type of relay optics system may be used andis not limited to two collimators. Furthermore, because the scanningsystem 20 has a coaxial architecture, both MEMS mirrors 21 and 24 actboth as transmission mirrors and a receiver mirrors.

A MEMS mirror is a mechanical moving mirror (i.e., a MEMS micro-mirror)integrated on a semiconductor chip (not shown). The MEMS mirrors may besuspended by mechanical springs (e.g., torsion bars) or flexures and isconfigured to rotate about a single axis and can be said to have onlyone degree of freedom for movement. For example, MEMS mirror 21 may beconfigured to oscillate on an x-axis to perform horizontal scanning,whereas MEMS mirror 24 may be configured to oscillate on a y-axis (i.e.,orthogonal to the x-axis) to perform vertical scanning. Due to thissingle axis of rotation, a MEMS mirror is referred to as a 1D MEMSmirror. Together, the two MEMS mirrors 21 and 24 are able to performscanning in two-dimensions (2D) and may be used for Lissajous scanningoperations.

Because the scanning system 20 is a 2D scanning system, the light beamshave a dot shape that are transmitted into a user's field of view, withhigher resolution being directed towards the retina of a user, and, moreparticularly, towards the fovea of the user.

A MEMS mirror itself is a non-linear resonator (i.e., a resonant MEMSmirror) configured to oscillate “side-to-side” about a scanning axissuch that the light reflected from the MEMS mirror oscillates back andforth in a scanning direction (e.g., a horizontal scanning direction ora vertical scanning direction). A scanning period or an oscillationperiod is defined, for example, by one complete oscillation from a firstedge of a field of view (e.g., first side) to a second edge of the fieldof view (e.g., second side) and then back again to the first edge. Amirror period of a MEMS mirror corresponds to a scanning period.

Thus, the field of view is scanned in both scanning directions bychanging the angle θ of each MEMS mirror on its respective scanningaxis. For example, the MEMS mirror 21 may be configured to oscillate ata resonance frequency of 2 kHz at a predetermined angular range to steerthe light over a scanning range. Thus, the field of view may be scanned,line-by-line, by a rotation of the MEMS mirror through its degree ofmotion. One such sequence through the degree of motion (e.g., from −15degrees to +15 degrees) is referred to as a single scan or scanningcycle.

Alternatively, it will be further appreciated that it is also possiblethat one of the MEMS mirrors 21 or 24 is configured as a 2D MEMS mirrorhaving two scanning axes (i.e., an x-axis and a y-axis) and the othermirror is a configured as a fixed reflective structure. For example,MEMS mirror 21 may be replaced with a fixed reflective structure andMEMS mirror 24 may be configured as a 2D MEMS mirror whose deflectionposition is controlled by driving the mirror on two axes. In this case,both axes of a single 2D MEMS mirror are controlled by a differentphase-locked loops (PLLs) such that a first scanning direction accordingto a first axis and a second scanning direction according to a secondaxis are separately controllable in terms of both driving amplitude anddriving frequency. Or the system can even be built without mirror 21 byrelocating the RGB unit 10 in a way to direct the light beam onto the 2Dscanning mirror 24.

The same can be said when using two 1D MEMS mirrors—both MEMS mirrorsare separately controllable in terms of both driving amplitude anddriving frequency. In this example, the scanning system 20 includes twoMEMS drivers 61 and 64 that are configured to drive a respective one ofMEMS mirror 21 and MEMS mirror 24 according to a respective drivingwaveform. The deflection angle of each MEMS mirrors 21 and 24continuously varies over time based on its driving waveform.Alternatively, circuitry for driving each MEMS mirror may be combinedinto a single MEMS driver (e.g., comprising two PLL circuits).Therefore, it will be understood that any functionality performed byeither of the MEMS drivers 61 and 64 may also be performed by a singleMEMS driver.

The scanning frequency of MEMS mirror 21 may be set higher than thescanning frequency of MEMS mirror 24. For this reason, MEMS mirror 21may be referred to as a “fast” mirror and MEMS mirror 24 may be referredto as a “slow” mirror. Additionally, the deflection angle (i.e., thetilt angle) of each mirror may follow a different pattern. For example,the deflection angle of MEMS mirror 21 may follow a sinusoidal patternat a higher frequency and the deflection angle of MEMS mirror 24 mayfollow a saw-tooth pattern at a lower frequency. This results in theMEMS mirror 21 having a faster rate of change about its scanning axis incomparison to MEMS mirror 24. Additionally, the deflection angle of oneor both mirrors may be modulated based on a detected and tracked foveaposition, to be described in more detail below.

As noted above, the RGB light unit 10 transmits the red, green, and bluelight beams towards a scanning system 20. The RGB light beams may becoupled into a transmission path via respective optical beam splitters11R, 11G, and 11B that have a high reflectivity with to their respectiveRGB light beam. The RGB light beams may pass through an optical beamsplitter 12 that has a high transmittivity with respect to the RGB lightbeams. The scanning system 20 is configured to receive the RGB lightbeams and steer the RGB light beams in a 2D projection plane using thetwo scanning axes to create an RGB image.

In particular, the scanning system 20 directs the RGB light at awaveguide 30 comprising a couple-in grating 32 and a couple-out grating34. The couple-in grating 32 couples light (e.g., RGB light and IRlight) into the waveguide 30. The coupled-in light travels along thewaveguide 30 via internal refraction towards the couple-out grating 34,which couples out the light. The couple-out grating 34 projects thecoupled-out light into a field of view of a user's eye, and, moreparticularly, projects the coupled-out light onto the user's eye. Thus,the waveguide 30 is responsible for coupling in an RGB image formed byRGB light and then projecting the RGB image into an eye of a user bycoupling out the RGB image into the field of view of the user's eye. Inother words, the waveguide 30 delivers RGB images generated by the RGBlight unit 10 to the user's eye in accordance with a controlled systemresolution that is adjusted based on the detected and tracked foveaposition.

The image projection system 100 further includes an eye-tracking sensor40 that, together with the scanning system 20, forms a retina scanner.The eye-tracking sensor 40 includes an IR light source 41 (i.e., asecond transmitter), such as a laser diode or a light emitting diode,that generates and transmits IR light beams. The IR light source 41 maybe a Near Infrared (NIR) laser source that generates laser pulses in thenear-infrared region of the electromagnetic spectrum (e.g., from 780 nmto-1350 nm). The sensor 40 further includes an optical beam splitter 42that has a high transmittivity with respect to the IR light beams. Theoptical beam splitter 12 is configured to receive the IR light beamsfrom the IR light source 41, and couple the IR light beams via highreflectivity into the transmission path. The scanning system 20,arranged on the transmission path, receives both RGB light beams and theIR light beams. With regard to the sensor function, the scanning system20 applies the 2D scanning function of MEMS mirrors 21 and 24 to the IRlight beams to scan a retina of a user's eye. This scanning occurssimultaneously, and thus in parallel, to the projection of the RGB imageonto the user's eye.

The IR light beams are reflected back from the user's eye, reflected inpart by the retina and further in part by the fovea. The backscatteredIR light is coupled back into the waveguide 30 and guided back to thesensor 40 to a light detector 43 (e.g., a photodetector such as aphotodiode). The waveguide 30, the scanning system 20, and the opticalbeam splitter 12 are arranged on a return path (i.e. a receive path) ofthe backscattered IR light that represents sensor data of the user'seye. In particular, the sensor data can be used by a signal processor togenerate an image of a user's eye, and specifically of a user's retina.

The optical beam splitter 12 has a high reflectivity for reflecting thebackscattered IR light and directing it towards the optical beamsplitter 42. Likewise, the optical beam splitter 42 has a highreflectivity for reflecting the backscattered IR light and directing ittowards the light detector 43. The light detector 43 is configured toreceive the backscattered IR light and generate electrical signals inresponse thereto. Since the time of transmission of each light pulsefrom the IR light source 41 is known, and because the light travels at aknown speed, a time-of-flight computation using the electrical signalscan determine the distance of objects from the light detector 43. Adepth map of a user's eye can be generated from the distanceinformation. A 2D reflectivity map (image of the retina) can begenerated by detecting the amplitude of electrical signals generated bylight reflected at the retina.

For example, the sensor 40 includes sensor circuitry 44 including an ADC45 and a digital signal processor (DSP) 46. The ADC 45 may be used forsignal detection and ToF measurement. For example, an ADC 45 may be usedto detect an analog electrical signal from the light detector 43 toestimate a time interval between a start signal (i.e., corresponding toa timing of a transmitted light pulse) and a stop signal (i.e.,corresponding to a timing of receiving an analog electrical signal atthe ADC 45) with an appropriate algorithm. The DSP 46 is configured toreceive the digital signals from the ADC 45 and preform single-channeldata processing to generate a retina image and then further performimage processing to perform eye-tracking.

The DSP 46 determines the time-of-flight and thus the distanceinformation corresponding to each IR light pulse transmitted by the IRlight source 41. Using position information corresponding to each MEMSmirror 21 and 24 (i.e., an x-y coordinate of transmitted IR lightprojected in 2D space onto a user's eye), the DSP 46 can map the depthand position of each transmitted IR light pulse to generate a depth map,and, more particularly, a retina image. The DSP 46 is further configuredto analyze the retina image to detect the focal direction of the eye.More specifically, the DSP 46 is configured to analyze the retina imageto detect and track the fovea within the retina image. The DSP 46 isconfigured to generate tracking information, including retina trackinginformation and/or fovea tracking information, and provide the trackinginformation to a system controller 50. The tracking information mayinclude real-time position information of the fovea within the retinaimage. The fovea position information may be in the form of x-ycoordinates, an identified region-of-interest (ROI)) within the retinaimage, an x-axis angular range (e.g., corresponding to MEMS mirror 21)in which the fovea is located, and/or a y-axis angular range (e.g.,corresponding to MEMS mirror 24) in which the fovea is located. Theysystem controller 50 is configured to perform one or more controlfunctions based on the received tracking information.

The system controller 50 is configured to control components of theimage projection system 100, including control of the IR light source41, the RGB light sources of the RGB light unit 10, and the MEMS drivers61 and 64. Thus, the system controller 50 includes control circuitry,such as a microcontroller, that is configured to generate controlsignals. In some examples, the system controller 50 may incorporate theDSP 46, a portion thereof, or may include additionally processingcircuitry for generating and/or analyzing retina image data, tracking afovea, and the like. The control signals may be used to control afunction of the transmitter 10 (i.e., a function of the RGB lightsources 11), a timing of firing light pulses by the IR light source 41,the oscillation frequency, oscillation pattern, and the oscillationrange (angular range of motion) of the MEMS mirrors 21 and 24, and/or adriving waveform of the MEMS drivers 61 and 64. Thus, the systemcontroller 50 may include at least one processor and/or processorcircuitry (e.g., comparators and digital signal processors (DSPs)) of asignal processing chain for processing data, as well as controlcircuitry, such as a microcontroller, that is configured to generatecontrol signals.

MEMS drivers 61 and 64 are configured to drive MEMS mirror 21 and MEMSmirror 24, respectively. In particular, each MEMS driver 61, 64 actuatesand senses the rotation position of its MEMS mirror about its scanningaxis, and provides position information (e.g., tilt angle or degree ofrotation about the scanning axis) of the mirror to the system controller50. Thus, each MEMS driver 61, 64 includes a measurement circuitconfigured to measure the rotation position of its MEMS mirror 21, 24.

For example, an actuator structure that is used to drive a respectiveMEMS mirror 21, 24 may be a comb-drive rotor and stator that include twodrive capacitors whose capacitance or stored charge is deflection angledependent. Thus, the measurement circuit may determine the rotationposition by measuring the capacitances of the drive capacitors or theirstored charges.

Based on this position information, the system controller 50 may controlone or more system functions in combination with the trackinginformation received from the DSP 46. The controllable functions basedon fovea detection and tracking will now be described in more detail.

FIGS. 2A and 2B illustrate two Lissajous scanning patterns in a 2Dscanning plane according to one or more embodiment. The 2D scanningplane is defined by an angular range of the MEMS mirror 21 between twoangular extrema in the horizontal scanning direction (e.g., between aleft edge and a right edge defined in the x-direction) and by an angularrange of the MEMS mirror 24 between two angular extrema in the verticalscanning direction (e.g., between a top edge and a bottom edge definedin y-direction). IR light and RGB light are projected onto an eyeaccording to preprogrammed scanning pattern (e.g., a raster or aLissajous pattern), where light pulses track the pattern.

FIG. 2A corresponds to a first scan of an eye (i.e., of a retina) usedfor eye detection and fovea detection. Here, scanning pattern is uniformsuch that the sine wave pattern is uniform in the y-direction indicatinga constant rate of change (i.e., oscillation speed) in the y-direction.Upon detecting a location of the fovea, the system controller 50 isconfigured to switch to foveated scanning, represented by FIG. 2B. Thus,FIG. 2B corresponds to a subsequent scan initialized once the locationof the fovea is determined.

In FIG. 2B, the scanning pattern is modified in the y-direction suchthat the pattern density of the scan is increased in a region of the 2Dscanning plan that corresponds with the determined fovea location. Tomodulate the scanning pattern, the system controller adjusts the drivingwaveform of MEMS driver 64 that drives MEMS mirror 24. For example, therate of change in the y-direction may be decreased in the y angularrange that corresponds to the location of the fovea such that a highernumber of scans (oscillations) is performed by MEMS mirror 21 in thatregion. This effectively increases the scanning density (i.e.,resolution) in that y angular range.

Since the IR scanner and the RGB imager share the same scanning system20, increasing the scanning density also increases the image resolutionof the RGB image projected into the user's eye. Thus, projection of theRGB image is rendered accordingly. Increasing the scanning density orresolution also enhances fovea detection and tracking by the sensor 40,thereby increasing the precision of the localizing of the fovea.

Additionally, the rate of change in the y-direction may be increased inthe y angular range that corresponds to locations located away from ordistant to the fovea such that a lower number of scans (oscillations) isperformed by MEMS mirror 21 in that region. This effectively decreasesthe scanning density (i.e., resolution) in that y angular range. Sincethe IR scanner and the RGB imager share the same scanning system 20,decreasing the scanning density also decreases the image resolution ofthe RGB image projected into the user's eye. Thus, projection of the RGBimage is rendered accordingly.

The scanning density may be gradually increased from a region locateddistant to the fovea as the scanning coordinates move towards a focusarea in which the fovea is located, and gradually decreased as thescanning coordinates move away from the focus area in which the fovea islocated.

It is noted that the scanning pattern of MEMS mirror 21 remains fixed(i.e., the MEMS mirror's driving waveform is fixed) while the scanningpattern of MEMS mirror 24 is adjusted based on fovea location. However,it will be further appreciated that the scanning pattern of MEMS mirror21 (e.g. its driving waveform) can also be modulated by the systemcontroller 50 to further define the focus area of the scanning pattern.

Accordingly, the system controller 50 is configured to track the foveabased on tracking information provided by the DSP 46 and adjust a focusarea of the scanning pattern in which the scanning density is increasedsuch that the focus area is defined in the scanning pattern. Thelocation of the focus area is adjusted to coincide with the trackedlocation of the fovea. The focus area is adjusted in real-time to followany changes in location of the fovea. For example, the focus area may beshifted up or down in the y-direction based on the detected fovealocation by adjusting the driving waveform of MEMS mirror 24. Thescanning pattern can be adjusted after each scan such that the scanningpattern for the next scan is updated based on the detected fovealocation. Alternatively, the scanning pattern can be adjusted mid-scanin response to detecting the location of the fovea. For example, thedriving waveform of the MEMS mirror 24 can be adjusted on aperiod-by-period basis.

Additionally, or alternatively, the system controller 50 may adjust theresolution of the RGB image by modulating the RGB light pulses to rendera higher resolution in the fovea region of the RGB image and to render alower resolution in the RGB image in areas outside of the fovea region.The RGB light pulses may be modulated in pulse width, in transmissionfrequency (i.e., the frequency in timing of light pulses—more frequentor less frequent), in brightness, or any combination thereof. To avoidconfusion with spectrum frequency, transmission frequency may bereferred to as pulse rate.

FIG. 3A shows MEMS mirror driving waveforms for Lissajous scanningaccording to one or more embodiments. In particular, the top waveform isa driving waveform for MEMS mirror 21 for horizontal scanning in thex-direction and the bottom waveform is a driving waveform for MEMSmirror 24 for vertical scanning in the y-direction. As can be seen, theX driving waveform of MEMS mirror 21 is sinusoidal and has a higheroscillation rate compared to the Y driving waveform of MEM mirror 24,which has a saw-tooth waveform. Accordingly, the MEMS mirror 21oscillates a plurality of times within an oscillation (scanning) periodof MEMS mirror 24.

FIG. 3B shows a Lissajous scanning pattern generated based on the MEMSmirror driving waveforms shown in FIG. 3A. In addition, RGB laser pulsesare shown in the 2D projection plane overlain on a portion of thescanning pattern. The system controller 50 is configured to modulate thepulse width, the pulse rate, and/or a laser driving current bandwidth ofthe RGB laser pulses depending on whether the RGB pulses are beingtransmitted within the fovea region or outside of the fovea region. Thepower of the RGB lasers can also be modulated, with higher brightness(i.e., higher power) being triggered by the system controller 50 forpulses fired within the fovea region and lower brightness beingtriggered by the system controller 50 for pulses fired outside of thefovea region. The laser driving current bandwidth is the frequency rangethat can be supported without having a significant change in the outputand is dependent on the analog modulation method applied to the lightsource—where analog modulation means that the waveform is continuouslyvarying in amplitude.

The fovea region of the RGB image is an ROI that coincides with thedetected location of the fovea, and may also be referred to as a focusregion. In this case, the system controller 50 may reduce a pulse widthof the RGB light pulses, increase their pulse rate, and/or increasetheir laser driving current bandwidth when transmitting the fovea regionof the RGB image. Brightness could also be increased. Thesereduced-width, higher pulse rate, higher current bandwidth RGB lightpulses are transmitted based on a scanning positions of MEMS mirrors 21and 24. For instance, reduced-width, higher pulse rate RGB light pulsesor reduced-width, higher current bandwidth RGB light pulses aretransmitted when the angular position of MEMS mirror 21 and the angularposition of MEMS mirror 24 about their respective scanning axes matchthe x-y coordinates of the fovea region. Thus, the RGB image has ahigher resolution in an area that is projected onto the fovea, and has alower resolution in areas that are projected onto other areas of theeye.

Different discrete levels of resolution may be defined in this manner.For example, a highest resolution may be rendered in the fovea region, asecond intermediate resolution may be rendered (e.g., via the anintermediate pulse width, intermediate brightness, an intermediate pulserate, and/or intermediate laser driving current bandwidth) in a regionadjacent to and concentric with the fovea region, and a lowestresolution may be rendered (e.g., via the largest pulse width, lowestbrightness, the lowest transmission pulse rate, and/or the lowest laserdriving current bandwidth) at a peripheral region of the retina. Thus,any combination of pulse width, pulse brightness, pulse rate, and laserdriving current bandwidth may be used to define different discretelevels of RGB image resolution. The system controller 50 adjusts bothparameters via control signals sent to the RGB light unit 10 byreferring both to the MEMS mirror position information and the foveatracking information.

Thus, eye tracking/fovea tracking via the DSP 46 defines the location ofa high resolution (foveated) area, and the horizontal resolution can betuned by a laser-pulse width. In addition, the brightness of a pixel canbe independently set by adjusting a peak current, a bandwidth of thecurrent, and/or a duty cycle for each RGB light source.

FIG. 4A shows MEMS mirror driving waveforms for Lissajous scanningaccording to one or more embodiments. In particular, the top waveform isa driving waveform for MEMS mirror 21 for horizontal scanning in thex-direction and the bottom waveform is a driving waveform for MEMSmirror 24 for vertical scanning in the y-direction. As can be seen, theX driving waveform of MEMS mirror 21 is sinusoidal and has a higheroscillation rate compared to the Y driving waveform of MEMS mirror 24,which has an adjustable saw-tooth waveform. Accordingly, the MEMS mirror21 oscillates a plurality of times within an oscillation (scanning)period of MEMS mirror 24. Here, the slope (i.e., rate of change) of theY driving waveform is adjusted based on the location of the fovearegion. This results in an adjustment to the rotational speed of MEMSmirror 24. FIG. 4B shows a Lissajous scanning pattern generated based onthe MEMS mirror driving waveforms shown in FIG. 4A.

According to this example, the slope of the rising edge of the Y drivingwaveform is adjusted to be higher when the scanning pattern is outsideof the fovea region and lower when the scanning pattern is inside of oroverlaps with the fovea region. Thus, the system controller 50 decreasesthe slope of the Y driving waveform, thereby slowing down the rotationalmovement of the MEMS mirror 24 when its angular position about itsscanning axis coincides with the angular range of the fovea region. Incontrast, the X driving waveform remains fixed. As a result, a highernumber of oscillations are performed by the MEMS mirror 21 relative tothe angular change of MEMS mirror 24, thereby creating a focus region inthe scanning pattern having a higher pattern density. The higher patterndensity produces a higher resolution for that region of the RGB image.

In addition, the system controller 50 increases the slope of the Ydriving waveform when the angular position of the MEMS mirror 24coincides with the angular range of a peripheral region of the retina.As a result, a lower number of oscillations are performed by the MEMSmirror 21 in this region, thereby creating a region in the scanningpattern having a lower pattern density. The lower pattern densityproduces a lower resolution for that region of the RGB image. Differentdiscrete levels of resolution may be defined in this manner.

The system controller 50 may also adjust the emission timing of the RGBlight pulses, with an increased pulse rate within the focus region ofthe scanning pattern and a decreased pulse rate outside of the focusregion of the scanning pattern. It is noted that the refresh rate andthe total number of scan lines is unchanged. Instead, the number of scanlines is compressed in the focus region and more widely spread apartoutside of the focus region. The spacing between scan lines maygradually increase with distance from a center of the focus region(i.e., a center of the fovea) or may be fixed within the focus regionand may gradually increase with distance from top and bottom edges ofthe focus region.

The system controller 50 may also adjust the intensity or brightness ofthe RGB light pulses based on similar criteria described above.

In view of the above, the eye-tracking system within the imageprojection system 100 can be used for dynamic foveated rendering thatfollows the user's gaze using eye tracking, and, more specifically,fovea tracking, and a renders a sharp image wherever the user's retinasare looking. A highest resolution of an image is rendered onto the foveausing the above-described techniques, whereas lower resolutions arerenders outside of the fovea region. Such a system may be used in XRsystems where the waveguide 30 is a smart eye-glass, a smart contactlens, a head-up display (HUD), a head-mounted display (HMDs), or thelike.

Although embodiments described herein relate to MEMS devices with amirror, it is to be understood that other implementations may includeoptical devices other than MEMS mirror devices. In addition, althoughsome aspects have been described in the context of an apparatus, it isclear that these aspects also represent a description of thecorresponding method, where a block or device corresponds to a methodstep or a feature of a method step. Analogously, aspects described inthe context of a method step also represent a description of acorresponding block or item or feature of a corresponding apparatus.Some or all of the method steps may be executed by (or using) a hardwareapparatus, like for example, a microprocessor, a programmable computeror an electronic circuit. In some embodiments, some one or more of themethod steps may be executed by such an apparatus.

While various embodiments have been described, it will be apparent tothose of ordinary skill in the art that many more embodiments andimplementations are possible within the scope of the disclosure.Accordingly, the invention is not to be restricted except in light ofthe attached claims and their equivalents. With regard to the variousfunctions performed by the components or structures described above(assemblies, devices, circuits, systems, etc.), the terms (including areference to a “means”) used to describe such components are intended tocorrespond, unless otherwise indicated, to any component or structurethat performs the specified function of the described component (i.e.,that is functionally equivalent), even if not structurally equivalent tothe disclosed structure that performs the function in the exemplaryimplementations of the invention illustrated herein.

Furthermore, the following claims are hereby incorporated into thedetailed description, where each claim may stand on its own as aseparate example embodiment. While each claim may stand on its own as aseparate example embodiment, it is to be noted that—although a dependentclaim may refer in the claims to a specific combination with one or moreother claims—other example embodiments may also include a combination ofthe dependent claim with the subject matter of each other dependent orindependent claim. Such combinations are proposed herein unless it isstated that a specific combination is not intended. Furthermore, it isintended to include also features of a claim to any other independentclaim even if this claim is not directly made dependent to theindependent claim.

It is further to be noted that methods disclosed in the specification orin the claims may be implemented by a device having means for performingeach of the respective acts of these methods.

Further, it is to be understood that the disclosure of multiple acts orfunctions disclosed in the specification or in the claims may not beconstrued as to be within the specific order. Therefore, the disclosureof multiple acts or functions will not limit these to a particular orderunless such acts or functions are not interchangeable for technicalreasons. Furthermore, in some embodiments a single act may include ormay be broken into multiple sub acts. Such sub acts may be included andpart of the disclosure of this single act unless explicitly excluded.

Instructions may be executed by one or more processors, such as one ormore central processing units (CPU), digital signal processors (DSPs),general purpose microprocessors, application specific integratedcircuits (ASICs), field programmable logic arrays (FPGAs), or otherequivalent integrated or discrete logic circuitry. Accordingly, the term“processor” or “processing circuitry” as used herein refers to any ofthe foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated hardware and/or software modules. Also, the techniques couldbe fully implemented in one or more circuits or logic elements.

Thus, the techniques described in this disclosure may be implemented, atleast in part, in hardware, software, firmware, or any combinationthereof. For example, various aspects of the described techniques may beimplemented within one or more processors, including one or moremicroprocessors, DSPs, ASICs, or any other equivalent integrated ordiscrete logic circuitry, as well as any combinations of suchcomponents.

A controller including hardware may also perform one or more of thetechniques described in this disclosure. Such hardware, software, andfirmware may be implemented within the same device or within separatedevices to support the various techniques described in this disclosure.Software may be stored on a non-transitory computer-readable medium suchthat the non-transitory computer readable medium includes a program codeor a program algorithm stored thereon which, when executed, causes thecontroller, via a computer program, to perform the steps of a method.

Although various exemplary embodiments have been disclosed, it will beapparent to those skilled in the art that various changes andmodifications can be made which will achieve some of the advantages ofthe concepts disclosed herein without departing from the spirit andscope of the invention. It will be obvious to those reasonably skilledin the art that other components performing the same functions may besuitably substituted. It is to be understood that other embodiments maybe utilized and structural or logical changes may be made withoutdeparting from the scope of the present invention. It should bementioned that features explained with reference to a specific figuremay be combined with features of other figures, even in those notexplicitly mentioned. Such modifications to the general inventiveconcept are intended to be covered by the appended claims and theirlegal equivalents.

What is claimed is:
 1. An image projection system, comprising: a firsttransmitter configured to generate pixel light pulses and transmit thepixel light pulses along a transmission path to be projected onto an eyeto render a projection image thereon; a second transmitter configured togenerate infrared (IR) light pulses transmitted along the transmissionpath and to be projected onto the eye and reflected back therefrom asreflected IR light pulses on a reception path; a coaxial scanning systemarranged along the transmission path and the reception path, the coaxialscanning system comprising at least one oscillator structure thatenables the coaxial scanning system to steer the pixel light pulses andthe IR light pulses in a first scanning direction and in a secondscanning direction according to a scanning pattern; an eye-trackingsensor configured to receive the reflected IR light pulses from thecoaxial scanning system, generate a retina image of the eye based on thereflected IR light pulses, and process the retina image to determine alocation of a fovea region of the eye; and a system controllerconfigured to render the projection image based on the location of thefovea region, wherein the projection image is rendered with a higherresolution in the fovea region and is rendered with a lower resolutionoutside of the fovea region.
 2. The image projection system of claim 1,wherein the coaxial scanning system is configured to steer the pixellight pulses in the first scanning direction and in the second scanningdirection to render the projection image, and to steer the IR lightpulses in the first scanning direction and in the second scanningdirection to perform retina scanning.
 3. The image projection system ofclaim 1, wherein the system controller is configured to control thefirst transmitter to decrease a pulse width and increase at least one ofa pulse rate or a laser driving current bandwidth for a first portion ofthe pixel light pulses projected onto the fovea region, and control thefirst transmitter to increase the pulse width and decrease at least oneof the pulse rate or the laser driving current bandwidth for at least asecond portion of the pixel light pulses projected outside of the fovearegion.
 4. The image projection system of claim 1, wherein the systemcontroller is configured to control the first transmitter to increase anintensity of the pixel light pulses for a first portion of the pixellight pulses projected onto the fovea region, and control the firsttransmitter to decrease the intensity of the pixel light pulses for atleast a second portion of the pixel light pulses projected outside ofthe fovea region.
 5. The image projection system of claim 4, wherein thefirst transmitter comprises a plurality of monochromatic light sourcesand the system controller is configured to control the intensity of thepixel light pulses by independently adjusting a peak and a bandwidth ofcurrent or a duty cycle of each of the plurality of monochromatic lightsources.
 6. The image projection system of claim 1, wherein the systemcontroller is configured to control the coaxial scanning system toincrease a pattern density of the scanning pattern in a first region ofthe scanning pattern that corresponds with the fovea region, and tocontrol the coaxial scanning system to decrease the pattern density ofthe scanning pattern in at least a second region of the scanning patternthat does not correspond with the fovea region.
 7. The image projectionsystem of claim 1, wherein the first transmitter is a red-green-blue(RGB) transmitter comprising a red light source, a green light source,and a blue light source, and the projection image is an RGB image. 8.The image projection system of claim 1, wherein each of the at least oneoscillator structure has at least one deflection angle that continuouslyvaries over time.
 9. The image projection system of claim 1, wherein theat least one oscillator structure comprises: a first oscillatorstructure configured to oscillate about a first scanning axis at a firstscanning frequency to steer the pixel light pulses and the IR lightpulses in the first scanning direction; and a second oscillatorstructure configured to oscillate about a second scanning axis at asecond scanning frequency to steer the pixel light pulses and the IRlight pulses in the second scanning direction.
 10. The image projectionsystem of claim 9, wherein the first scanning frequency is greater thanthe second scanning frequency.
 11. The image projection system of claim9, wherein the coaxial scanning system comprises: a first driver circuitconfigured to drive the first oscillator structure according to a firstdriving waveform that is fixed; and a second driver circuit configuredto drive the second oscillator structure according to a second drivingwaveform that is adjustable.
 12. The image projection system of claim11, wherein the first driving waveform is a sine wave and the seconddriving waveform is a saw-tooth wave.
 13. The image projection system ofclaim 11, wherein the system controller is configured to adjust thesecond driving waveform based on the location of the fovea region. 14.The image projection system of claim 13, wherein the system controlleris configured to decrease a slope of the second driving waveform when adeflection angle of the second oscillator structure is within an angularrange that corresponds to the location of the fovea region, and increasethe slope of the second driving waveform when the deflection angle ofthe second oscillator structure is outside the angular range thatcorresponds to the location of the fovea region.
 15. The imageprojection system of claim 9, wherein the system controller isconfigured to dynamically adjust a rotational speed of the secondoscillator structure in real-time based on the location of the fovearegion.
 16. The image projection system of claim 15, wherein the systemcontroller is configured to adjust the rotational speed of the secondoscillator structure such that a pattern density of the scanning patternis increased in a first region of the scanning pattern that correspondswith the fovea region and decreased in at least a second region of thescanning pattern that does not correspond with the fovea region.
 17. Theimage projection system of claim 16, wherein the system controller isconfigured to decrease the rotational speed of the second oscillatorstructure in the first region of the scanning pattern that correspondswith the fovea region and increase the rotational speed of the secondoscillator structure in at least a second region of the scanning patternthat does not correspond with the fovea region.
 18. A method ofprojecting an image based on fovea tracking, the method comprising:transmitting pixel light pulses along a transmission path to beprojected onto an eye to render a projection image thereon; transmittinginfrared (IR) light pulses along the transmission path, the IR lightpulses being projected onto the eye and reflected back therefrom asreflected IR light pulses on a reception path; steering the pixel lightpulses and the IR light pulses in a first scanning direction and in asecond scanning direction according to a scanning pattern; sensing thereflected IR light pulses received from the reception path; generating aretina image of the eye based on the reflected IR light pulses;processing the retina image to determine a location of a fovea region ofthe eye; and rendering the projection image based on the location of thefovea region, wherein the projection image is rendered with a higherresolution in the fovea region and is rendered with a lower resolutionoutside of the fovea region.
 19. The method of claim 18, whereinrendering the projection image based on the location of the fovea regioncomprises: decreasing a pulse width and increasing at least one of apulse rate or a laser driving current bandwidth for a first portion ofthe pixel light pulses projected onto the fovea region; and increasingthe pulse width and decreasing at least one of the pulse rate the laserdriving current bandwidth for at least a second portion of the pixellight pulses projected outside of the fovea region.
 20. The method ofclaim 18, wherein rendering the projection image based on the locationof the fovea region comprises: increasing an intensity of the pixellight pulses for a first portion of the pixel light pulses projectedonto the fovea region; and decreasing the intensity of the pixel lightpulses for at least a second portion of the pixel light pulses projectedoutside of the fovea region.
 21. The method of claim 18, whereinrendering the projection image based on the location of the fovea regioncomprises: increasing a pattern density of the scanning pattern in afirst region of the scanning pattern that corresponds with the fovearegion; and decreasing the pattern density of the scanning pattern in atleast a second region of the scanning pattern that does not correspondwith the fovea region.